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INTRODUCTION 

Within NASA’s recent thrust for industrial col- 
laboration, JPL (Jet Propulsion Laboratory) has 
recently established two technology cooperation 
agreements in the robotics area: one on virtual re- 
ality (VR) calibration with Deneb Robotics, Inc., 
and the other on redundant manipulator con- 
trol with Robotics Research Corporation (RRC). 
These technology transfer cooperation tasks will 
enable both Deneb and RRC to commercialize en- 
hanced versions of their products that will greatly 
benefit both space and terrestrial telerobotic ap- 
plications. 

COMMERCIALIZATION OF 
JPL VIRTUAL REALITY 
CALIBRATION TECHNOLOGY 

JPL recently developed a virtual reality (VR) 
calibration technique that enables reliable and ac- 
curate matching of a graphically simulated vir- 
tual environment in 3-D geometry and perspec- 
tive with actual video camera views [1], [2], This 
technique enables high-fidelity preview/predictive 
displays with calibrated graphic overlay on live 
video for telerobotic servicing applications. Its 
effectiveness was successfully demonstrated in a 
recent JPL (Jet Propulsion Laboratory)/NASA- 
GSFC (Goddard Space Flight Center) ORU (Or- 
bital Replacement Unit) changeout remote servic- 
ing task. The current JPL VR calibration is a 
two-step procedure: camera calibration followed 
by object localization. Key new features of this 


JPL VR calibration technique include: 1) an 

operator-interactive method adopted to obtain re- 
liable correspondence data, 2) a robot arm itself 
used as a calibration fixture for camera calibra- 
tion, eliminating a cumbersome procedure of us- 
ing external calibration fixtures, 3) the object lo- 
calization procedure added after the camera cal- 
ibration to obtain graphic overlay of both the 
robot arm and the object(s) on live video enabling 
effective use of the computer-generated trajectory 
mode in addition to the teleoperation mode, 4) 
a projection-based linear least-squares algorithm 
extended to handle multiple camera views for ob- 
ject localization, and 5) nonlinear least-squares al- 
gorithms combined with linear ones employed for 
both camera calibration and object localization. 
Details of the algorithms and their software list- 
ings [3] were prepared as part of this JPL- Industry 
cooperative task. 

An example of a calibrated graphic over- 
lay after the virtual reality calibration for the 
JPL/NASA-GSFC remote servicing demonstra- 
tion is shown in Figure 1. The positioning align- 
ment accuracy achieved in inserting a tool into 
the ORU hole using 4 camera views was 0.51 
cm on the average with a 1.07 cm maximum er- 
ror at 95% confidence level. After matching 3- 
D graphics models of a virtual environment with 
actual camera views through the above virtual 
reality calibration technique, the operator can 
now perform a telerobotic servicing task with pre- 
view/predictive displays having calibrated graph- 
ics overlay on live video. Preview/predictive dis- 
plays allow the operator to generate the simulated 
robot arm trajectory in preview and then to vi- 
sually monitor and verify the actual remote robot 
arm motion with confidence, and thus provide 
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Figure 1: Overlay of calibrated 3-D graphic models 
(wire-frames with semi-transparent surfaces) on live 
video for telerobotic satellite servicing. 


effective visual prediction/verification to the oper- 
ator and enhance safety and reliability in remote 
servicing operations regardless of communication 
time delay. Figure 2 shows a snapshot of a pre- 
view/predictive display during the performance of 
the JPL/GSFC demonstration. 

Approach 

We have taken the following approach in our 
JPL-Industry cooperative Deneb Commercializa- 
tion Task: 1) JPL transfers the VR calibration 
software technology to Deneb, 2) Deneb, coop- 
erating with JPL, inserts this software technol- 
ogy into its commercial product TELEGRIP as 
the video overlay /VR calibration option for mar- 
keting, and 3) in return, NASA utilizes this en- 
hancement of a commercially supported product 
for NASA applications. 

The virtual reality calibration option imple- 
mented on TELEGRIP will be an important el- 
ement to build a state-of-the art VR interface in 
telerobotic applications with preview/predictive 
displays. Thus, the enhanced Deneb product 
can be effectively used in both space and ter- 
restrial telerobotics applications, providing 1) im- 
mediate benefits to NASA for ground-controlled 
telerobotic servicing in space, 2) immediate bene- 
fits to the national DOE (Department of Energy) 
labs working on the disposal and remediation of 
nuclear waste, and 3) foreseeable potential ap- 
plications in automotive manufacturing, medical 
telerobotic surgery, telerobotic construction, and 
maintenance robots. 


Figure 2: A snapshot of a preview/predictive display 
during the performance of the ORU extraction in the 
JPL/GSFC ORU changeout demonstration task. 


Implementation on TELEGRIP 

The JPL virtual reality calibration option is 
currently being implemented on Deneb ’s TELE- 
GRIP [4] which is an open architecture based 
upon Dynamic Shared Objects (DSO’s). DSO’s 
provide many benefits when compared with other 
strategies for incorporating user-defined modules 
with a centralized kernel, including 1) speed of 
development, 2) access to all internal functions 
and data, including the entire geometric database, 
3) flexibility in development, and 4) minimizing 
platform dependence. A key important feature 
provided by this TELEGRIP open architecture is 
that it allows developers/users to add their own 
virtual reality calibration algorithms and video 
overlay methods, if necessary. 

Both one-window and two-window graph- 
ics/video displays are planned to be supported 
for VR calibration. Under the one-window cal- 
ibration strategy, the TELEGRIP graphics dis- 
play is divided into two separate vertically ar- 
ranged NTSC-size (National Television Systems 
Committee standard) viewports. One viewport 
contains the live video image of the work envi- 
ronment, while the other displays the equivalent 
3D graphical model. Upon completion of the cam- 
era calibration and object localization phases, the 
graphics-overlaid video image will be available to 
display in one of the viewports or to display on 
a separate NTSC monitor. The two-window ap- 
proach relies upon two external NTSC-size GL 
(Graphics Library) or GLX (Graphics Library in 
X environments) windows with one window con- 
taining the live video image and the other the 3D 
graphic display. This enables users to relocate the 
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windows in a manner desirable for their particular 
application. Upon completion of the camera cali- 
bration and object localization phases, a graphics- 
overlaid video image is available to display in any 
window, including the TELEGRIP window, or to 
display on a separate NTSC screen. 

The TELEGRIP video overlay implementation 
is based upon an application programmer inter- 
face (API) layer which insulates the overlay de- 
veloper from the specifics of video hardware, thus 
enabling support over a wide range of video prod- 
ucts. Support is currently planned for the SGI 
(Silicon Graphics, Inc.) VideoLab, Galileo, In- 
digo2, Indy, and Serius video boards encompass- 
ing the entire range of current SGI computing 
hardware from the Indy to the Onyx. Graphic 
models can be overlaid in wire- frame or in solid- 
shaded polygonal rendering, with varying levels of 
transparency to produce different visual effects. 

COMMERCIALIZATION OF 

JPL REDUNDANT MANIPULATOR 

CONTROL TECHNOLOGY 

Theoretical and experimental investigations 
have demonstrated that dexterous manipulation 
tasks can be carried out only by redundant, force- 
controlled robotic manipulators that possess flex- 
ibility and versatility comparable to the human 
arm. For research in this area, the Robotics Lab- 
oratory at JPL acquired in 1989 two redundant 7- 
DOF (degree- of- freedom) manipulators made by 
Robotics Research Corporation (RRC) of Ohio, 
the leading manufacturer of this type of manipu- 
lators since the mid 1980’s. 

At the time of purchase, neither the application 
domain nor the required redundant control laws 
for such advanced manipulators was fully devel- 
oped. JPL research has contributed to both areas 
by identifying tasks in which redundancy is es- 
sential and by developing an underlying control 
methodology for such manipulators. 

RRC has recently expanded and enhanced its 
product line by introducing a second-generation 
version of its manipulator that provides improved 
mechanical performance and employs a unique 
low-level control system in which all servo elec- 
tronics are mounted in the arm. It is now log- 
ical to begin integrating RRC’s state-of-the-art 
servomechanism technology with JPL’s advanced 
high-level control developments, and to prepare 
this new robot technology for commercial appli- 
cations. 

Under funding from NASA, the first phase of 
such a commercialization activity began in FY’94, 
with the transfer to RRC of an algorithm for re- 
dundant arm control developed at JPL[5-9] and 
widely used in the robotics community. This algo- 
rithm, known as Configuration Control, combines 


the specification of a set of constraint tasks with 
the end-effector prescribed trajectory to provide a 
highly efficient and powerful redundant arm con- 
trol strategy. 

Background 

During the course of the past two years, RRC 
has developed a unique servo control architecture 
for its manipulator arms which greatly reduces the 
need for expensive external power and computing 
electronics and replaces the costly internal arm 
wiring harness with a “fly-by- wire” data/power 
bus communication system. Miniature DSP (Dig- 
ital Signal Processor)- based servo control mod- 
ules, containing all computing and power elec- 
tronics, are collocated with the joint actuators 
in the manipulator arm joints. The parameters 
for the individual joint controllers are downloaded 
by a master computer via a high-speed commu- 
nication link. Since the remotely-located mas- 
ter computer is free from the burden of servo 
power and computing electronics, high-level con- 
trol functions can now be practically transferred 
to a general-purpose workstation or personal com- 
puter with significant cost savings. This new 
high-level RRC controller is designated the Next 
Generation Controller (RRC/NGC). 

In the area of redundant arm control, JPL has 
developed a class of motion control algorithms 
for redundant manipulators called Configuration 
Control (CC),[5-9]. In this approach, the user can 
specify task-dependent constraints for the redun- 
dant manipulator which have the effect of utilizing 
the robot redundancy and allowing efficient end- 
effector trajectory control. Since this approach 
was implemented originally on RRC manipula- 
tors and the resulting algorithms were extensively 
tested in several experiments, it is felt that this 
technology is mature enough to be transferred to 
industry and incorporated into RRC’s new product 
line (see Figure 3). 

The RRC/NGC system under development will 
be highly compatible with the kind of centralized 
high-level control embedded in the CC approach. 
The master computer used in the NGC system is 
a standard workstation, and it is well suited to 
run the CC algorithms. Furthermore the use of 
a workstation (or of a PC) as a master computer 
enables RRC to make use of enhanced graphic ca- 
pabilities to provide the user with a sophisticated 
interface for motion planning and control. 

Approach 

In order to ensure that the technology transfer 
proceeds smoothly, the following steps have been 
planned: 

1. Duplicate the hardware and software environ- 
ment of the RRC/NGC at JPL and test it with 
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Figure 3: 7-DOF Robotics Research arm. 


the RRC manipulators in the JPL Robotics Lab- 
oratory. 

2. Modify JPL Configuration Control algorithms 
to make them compatible with the NGC environ- 
ment, implement and test the algorithms on the 
master computer adopted in the NGC system and 
with the current RRC manipulators in the JPL 
Robotics Laboratory. 

3. Integrate the tested algorithms with the new 
RRC manipulators using the Next Generation 
Controller. 

Technology Transfer Issues 

A technology transfer task of this type requires 
the same steps as to transform a laboratory pro- 
totype into a commercial product. Once the func- 
tionality of the prototype, the CC algorithms 
in this case, has been established and verified, 
then the development efforts must focus on is- 
sues such as compatibility with the rest of the 
system, price/performance trade-off, documenta- 
tion, maintainability, and so on. 

The decision was made by RRC to implement 
as much as possible of their software in object- 
oriented format, and use an IBM-compatible per- 
sonal computer as the master controller. From the 
JPL side, it was necessary to re-engineer some ex- 
isting software to eliminate the dependency of the 
code on data structures related to the rest of the 
JPL system, and to port the programs to an op- 
erating system compatible with the IBM-PC that 
RRC has selected as its NGC platform. In the 


interest of compatibility with existing RRC soft- 
ware, as well as to minimize overall system cost, 
the real-time operating system selected is the In- 
tel iRMX running under Windows, which can ex- 
ecute RRC’s existing code as well as the new JPL 
Configuration Control software modules. 

The technology transfer is currently proceed- 
ing smoothly and most of the necessary programs 
have already been converted to a stand-alone con- 
figuration. We will be ready to integrate this soft- 
ware with the PC-based real-time system and test 
it with the RRC redundant manipulators in the 
JPL Robotics Laboratory later this year. 
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